On this page
reversing.kr - PEPassword
PEPassword
We are given 2 executables. Original.exe and Packed.exe. Running Original.exe displays a message box.

Running Packed.exe prompts for a password.

Packed.exe starts by loading kernel32.dll and user32.dll. It proceeds by creating a modeless dialog box by calling CreateDialogIndirectParamA and runs a standard
message loop. The message loop runs as long as the byte at ss:[ebp + 0x402A3E], which resolves to
0x409410, is 0.

If we manually set the value at that memory address using a debugger we will eventually reach this interesting set of instructions.

The function at 0x4091DA appears to be some sort of irreversible hashing function. The loop at 0x40921F appears to decrypt whatever is pointed by edi in
increments of 4 bytes. What is the value at edi? It's 0x401000, the beginning of the .text segment! If you look closely at the decryption loop, you can make an interesting observation:
the decryption relies only on the initial values of eax and ebx, but these values are the result of the hashing function. Luckily, we have access to the unpacked
binary, Original.exe. Thus, we know what the first 4 bytes should decrypt to: the first 4 bytes of the unpacked executable!

So we can get the initial value of eax by a simple xor operation. eax = 0x014cec81 ^ 0xb6e62e17. Where 0x014cec81 are the first 4 bytes of the .text segment of the
unpacked binary, and 0xb6e62e17 are the bytes of the packed binary. Getting the value of ebx won't be as easy, but we are in luck again, we have the power of
technology: the C compiler (or any other programming or scripting language of your choice).

Code
Since ebx is constrained by its effective width (0xffffffff), we can write a simple program imitating the decryption loop to determine which values of ebx result in correct decryption on the second iteration.
uint32_t rol(uint32_t value, uint8_t shift) {
return (value << shift) | (value >> (32 - shift));
}
uint32_t ror(uint32_t value, uint8_t shift) {
return (value >> shift) | (value << (32 - shift));
}
int main() {
uint32_t eax1 = 0x014cec81 ^ 0xb6e62e17; // eax in first iteration
uint32_t eax2 = 0x57560000 ^ 0x0d0c7e05; // desired eax in second iteration
printf("eax %#x\n", eax1);
for (uint32_t i = 0; i < 0xffffffff; i++) {
uint32_t eax = eax1;
uint32_t ebx = i;
ebx = rol(ebx, eax & 0xff);
eax ^= ebx;
eax = ror(eax, (ebx & 0x0000ff00) >> 8);
ebx = (ebx + eax) & 0xffffffff;
if (eax == eax2) {
printf("ebx: %#x\n", i);
}
}
}
Running the code gives us the following output:

We found 2 potential ebx values. Using a debugger, we can place a breakpoint at the decryption loop and manually set the registers
to the found values. Trying the first ebx value doesn't yield anything, as the program crashes from an illegal instruction in the .text section.
The second value however gets us the flag!
